Collecting and analyzing electronic survey responses including user-composed text

ABSTRACT

Embodiments of the present disclosure relate to collecting and analyzing electronic survey responses that include user-composed text. In particular, systems and methods disclosed herein facilitate collection of electronic survey responses in response to electronic survey questions. The systems and methods disclosed herein classify the electronic survey questions and determine a semantics model including customized operators for analyzing the electronic survey responses to the corresponding electronic survey questions. In addition, the systems and methods disclosed herein provide a presentation of the results of the analysis of the electronic survey responses via a graphical user interface of a client device.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.15/825,353, filed on Nov. 29, 2017. The aforementioned application ishereby incorporated by reference in its entirety.

BACKGROUND

Companies and other entities often rely on opinions and feedback fromcustomers, employees, or other individuals. A common method of acquiringfeedback is through electronic surveys, including electronic customerratings and reviews (e.g., ratings and reviews for products, services,businesses, etc.). For example, companies often administer electronicsurveys to customers to collect meaningful information about theexperience of any number of customers with a particular company orproduct. With the increased convenience and administration of electronicsurveys, companies can collect massive amounts of information and datafrom millions of customers.

Many conventional systems attempt to conserve processing resources bycollecting survey responses having an analysis-friendly format. Forexample, many electronic surveys include questions that solicitrankings, ranges of numbers, defined categories, binarycharacterizations, or other types of data that facilitate a less robustanalysis of the survey results. However, these types of electronicsurvey questions fail to gather valuable information from respondents asthe type of information that respondents provide is predefined. Indeed,if respondents have an issue that is not explicitly identified as achoice within a predefined choice, then that issue is almost impossibleto identify from an electronic survey.

Accordingly, most electronic survey administrators place a high value oncollecting free-form text responses from respondents that allow arespondent to provide information within the voice of the respondent. Asa result of the massive collection of text responses, however,conventional systems for analyzing the text-responses becomecomputationally bogged down and are computer resource intensive (e.g.,processor and memory resources) when attempting to identify specifictypes of information or sift through millions of text responses tosurvey questions to analyze trends of information within the responses.Indeed, conventional electronic survey systems often cannot providerobust analysis for free-form text responses, or alternatively, consumelarge amounts of computing resources to perform an analysis, which takessignificant computing time. Due to these limitations, conventionalsystems do not provide tools for determining overall trends orextracting meaningful information from the massive number of free-formtext responses.

BRIEF SUMMARY

Embodiments of the present disclosure provide benefits and/or solve oneor more of the foregoing or other problems in the art with systems,computer-readable media, and methods for collecting and analyzingelectronic survey responses that include user-composed text. Inparticular, systems and methods disclosed herein facilitate collectionof electronic survey responses to electronic survey questions. Thesystems and methods determine classifications for the electronic surveyquestions and associated semantics models for analyzing electronicsurvey responses to the electronic survey questions. In addition, thesystems and methods receive a search query requesting informationcontained within the electronic survey responses. The systems andmethods identify survey questions and responses that correspond to thesearch query and analyze the identified survey responses using anidentified semantics model uniquely configured to analyze responses tocorresponding survey questions. The systems and methods further providea presentation of results of the analysis of the survey responses.

Systems and methods described herein involve collecting and analyzingelectronic survey responses using features and functionality thatidentify information within user-composed text responses via anefficient utilization of processing resources. For example, the systemsand methods described herein generate, identify, or otherwise determinea semantics model including specific operators associated withidentifying certain types of information contained within electronicsurvey responses. In addition, the systems and methods described hereinselectively analyze electronic survey responses more likely to includeinformation requested by a search query. As described in further detailherein, the systems and methods include various features andfunctionality for identifying specific information contained withinelectronic survey responses while utilizing fewer processing resourcesthan conventional systems.

Additional features and advantages of the embodiments will be set forthin the description that follows, and in part will be obvious from thedescription, or may be learned by the practice of such exemplaryembodiments. The features and advantages of such embodiments may berealized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims. These, and otherfeatures, will become more fully apparent from the following descriptionand appended claims, or may be learned by the practice of such exemplaryembodiments as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the disclosure briefly described above will berendered by reference to specific embodiments thereof that areillustrated in the appended drawings. It should be noted that thefigures are not drawn to scale, and that elements of similar structureor function are generally represented by like reference numerals forillustrative purposes throughout the figures. Understanding that thesedrawings depict only typical embodiments of the disclosure and are nottherefore considered to be limiting of its scope, the disclosure will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings, in which:

FIG. 1 illustrates a block diagram of a survey analysis environment inaccordance with one or more embodiments;

FIG. 2 illustrates a flow diagram of interactions between devices of thesurvey system to collect and analyze electronic survey responses inaccordance with one or more embodiments;

FIG. 3 illustrates example semantic models associated with correspondingquestion classifications in accordance with one or more embodiments;

FIG. 4 illustrates an example operator of a semantic model includingexemplary operator functions in accordance with one or more embodiments;

FIGS. 5A-5C illustrate example graphical user interfaces for presentingresults of an analysis of electronic survey responses in accordance withone or more embodiments;

FIG. 6 illustrates a schematic diagram of a server device upon which anelectronic survey system is implemented in accordance with one or moreembodiments;

FIG. 7 illustrates a flowchart of a series of acts in a method forcollecting and analyzing electronic survey responses to an electronicsurvey question in accordance with one or more embodiments;

FIG. 8 illustrates a flowchart of a series of acts in a method forcollecting and analyzing electronic survey responses to a plurality ofelectronic survey questions in accordance with one or more embodiments;

FIG. 9 illustrates a block diagram of a computing device in accordancewith one or more embodiments; and

FIG. 10 illustrates a networking environment of an electronic surveysystem in accordance with one or more embodiments.

DETAILED DESCRIPTION

One or more embodiments described herein provide an electronic surveysystem that enables a user (e.g., an administrative user) to search acollection of electronic survey responses including user-composed textto identify information contained therein. In particular, the electronicsurvey system receives electronic survey responses corresponding toelectronic survey questions. The electronic survey system additionallyclassifies the electronic survey questions associated with the receivedelectronic survey responses. Based on the classification of theelectronic survey questions, the electronic survey system determines asemantics model for analyzing the corresponding electronic surveyresponses using operators configured to identify specific types ofinformation contained within the electronic survey responses. Inparticular, the electronic survey system can use the identifiedsemantics model to analyze the user-composed text of the electronicsurvey responses and provide the results of the analysis to the user.

As a general overview, the electronic survey system can collectelectronic survey responses from any number of respondents. In one ormore embodiments, electronic survey system administers an electronicsurvey including electronic survey questions delivered to respondentsvia associated respondent devices. The electronic survey system canadditionally collect electronic survey responses including user-composedtext (e.g., free-form, unstructured text responses) and store theelectronic survey responses on an electronic survey database.

In addition, before or after administering the electronic survey system,the electronic survey system can classify the electronic surveyquestion(s) of the electronic survey to determine a semantics model toapply to the user-composed text contained within correspondingelectronic survey responses. For example, in one or more embodiments,the electronic survey system determines that an electronic surveyquestion includes a request for a respondent to provide a particulartype of information (e.g., an opinion, a recommendation, a concern, acriticism, a complaint) within user-composed text. Based on a determinedtype of information requested by the electronic survey question, theelectronic survey system can identify a semantics model configured toanalyze user-composed text of the electronic survey responses to therespective electronic survey question and extract or otherwise identifythe requested type of information included within the correspondingelectronic survey responses.

In one or more embodiments, the electronic survey system furtherfacilitates a search of the electronic survey responses to identifyinformation contained therein. For example, in one or more embodiments,the electronic survey system receives a search query from a clientdevice (e.g., administrator device). In response to receiving the searchquery, the electronic survey system identifies one or more electronicsurvey questions having a question classification that corresponds toinformation requested by the search query and/or a topic identified bythe search query. In addition, the electronic survey system identifiesthose electronic survey responses for the identified one or moreelectronic survey questions and applies an associated semantics model tothe electronic survey responses to identify information requested by thesearch query. By selecting a semantics model based on a classificationfor the electronic survey question, the electronic survey systemanalyzes electronic survey response using a semantics model uniquelysuited or otherwise configured to identify the relevant information tothe search query.

Upon applying the semantics model to the relevant electronic surveyresponses, the electronic survey system can further generate and providea presentation of results of the analysis of the electronic surveyresponses. For example, the electronic survey system can identify andprovide a listing of any number of electronic survey responses thatinclude information identified using the semantics model. As anotherexample, the electronic survey system can extract words, phrases, andother information from the electronic survey responses and provide avisualization of the analysis results via a graphical user interface ofa client device. As will be described in further detail below, theelectronic survey system further provides various interactive featuresthat enable a user of the client device to modify the presentationand/or modify the semantics model for subsequently analyzing additionalelectronic survey responses.

As will be described in further detail below, the electronic surveysystem facilitates an improved approach to analyzing and searchingelectronic survey responses including user-composed text. As an example,the electronic survey system efficiently analyzes electronic surveyresponses on a response-by-response basis using a semantics modelcustomized for analyzing electronic survey responses corresponding tospecific electronic survey question. In particular, by classifyingelectronic survey questions that request a specific topic of information(or a specific type of information), the electronic survey system canmore efficiently analyze responses to the electronic survey questions byapplying a semantics model configured to identify the informationrequested by the corresponding electronic survey question(s) (andcorresponding to the information requested by the search query).

As another example, the electronic survey system more efficientlysearches a collection of responses to any number of electronic surveyquestions. For example, in one or more embodiments, the electronicsurvey system selectively analyzes electronic survey responses based ona classification of a corresponding electronic survey questions. In thisway, rather than analyzing all electronic survey responses that mayinclude a massive collection of user-composed text, the electronicsurvey system selectively identifies a subset of the electronic surveyresponses and analyzes those electronic survey responses predicted toinclude relevant information requested by a search query.

In addition, by identifying and applying semantics models uniquelyconfigured to analyze responses to a specific electronic survey question(or specific type of electronic survey question), the electronic surveysystem more accurately identifies relevant information contained withinthe user-composed text of electronic survey responses. In particular,applying a classification-specific semantics model enables theelectronic survey system to identify fewer false positives while missingfewer instances of relevant information than conventional methods ofsearching through electronic survey responses. In addition, as will bedescribed in further detail below, the electronic survey systemimplements various features and functionality to improve upon theaccuracy of semantics models over time for various electronic surveyquestions. Accordingly, features and functionality of the electronicsurvey system described herein accurately identify relevant informationwithin electronic survey responses while preserving processing resourcesof a system. In particular, the electronic survey system results inincreases computing efficiency, resulting in a significant reduction inprocessor and memory resources needed to provide a robust and qualityanalysis of free-form text responses.

As used herein, an “electronic survey” or “survey” refers to one or moreelectronic communications or an electronic document used to collectinformation about associated topics. In one or more embodiments,electronic surveys include electronic survey questions including arequest for information from one or more respondents (e.g., personresponding to an electronic survey) about a topic. As an example, anelectronic survey question can include a request for opinions,suggestions, questions, personal experiences, complaints, or other typesof information.

As used herein, an “electronic survey response,” “survey response,” or“response” refer interchangeably to any digital content received from arespondent in response to an electronic survey question. In one or moreembodiments described herein, an electronic survey response includesuser-composed text (e.g., free-form text) including words, phrases, andother content composed by a respondent based on a request forinformation or other open-ended question contained in an electronicsurvey question.

As used herein, a “semantics model” refers to a model includingoperators and operator functions for analyzing content of an electronicsurvey response. In particular, as will be described in further detailbelow, a semantics model includes operators for identifying particulartypes of information. For example, a semantics model can include one ormore operators configured to identify respondent opinions,recommendations, questions, and other types of information. In addition,as will be described in further detail below, a semantics model caninclude terminology information, including for example, identified termsand phrases associated with customized meanings within a context of atopic or type of information within a response.

Additional features and characteristics of the electronic survey systemare described below with respect to the figures. For example, FIG. 1illustrates a block diagram of a survey environment 100. In general, andas illustrated in FIG. 1, the survey environment 100 includes a serverdevice(s) 102 including an electronic survey system 104, which includesa response analysis system 106. The survey environment 100 furtherincludes respondent devices 108 a-n and associated respondents 110 a-n.As further shown, the survey environment 100 includes an administratordevice 112 and associated user 116 (e.g., an administrative user). Inone or more embodiments, the administrator device 112 further includes asurvey application 114 thereon.

As will be described in further detail below, the electronic surveysystem 104 implements various features and functionality describedherein with regard to administering electronic surveys, collectingelectronic survey responses, and analyzing electronic survey responses.As shown in FIG. 1, the electronic survey system 104 includes a responseanalysis system 106 that implements features and functionality describedherein with regard to classifying electronic survey questions,processing search queries requesting information contained within theelectronic survey questions, and providing a presentation of searchresults to the user 116 of the administrator device 112. Furthermore,while one or more embodiments described herein refer specifically to theelectronic survey system 104 performing features and functionalityrelated to implementing electronic surveys, implementing searchfeatures, and providing a presentation of search results, it will beunderstood that the response analysis system 106 can similarly performone or more of the features and functionality described herein.

As shown in FIG. 1, the server device(s) 102 can communicate with eachof the respondent devices 108 a-n and administrator device 112 over thenetwork 118, which may include one or multiple networks and may use oneor more communication platforms or technologies suitable fortransmitting data between devices. In one or more embodiments, thenetwork 118 includes the Internet or World Wide Web. In addition, or asan alternative, the network 118 can include various other types ofnetworks that use different communication technologies or protocols, asdescribed in further detail below.

Although FIG. 1 illustrates a particular number and arrangement of therespondent devices 108 a-n, the survey environment can nonethelessinclude any number of respondent devices 108 a-n. In addition, therespondent devices 108 a-n and administrator device 112 can refer tovarious types of computing devices. For example, one or more of thedevices 108 a-n, 112 may include a mobile device such as a mobile phone,a smartphone, a PDA, a tablet, or a laptop. Additionally, oralternatively, one or more of the devices 108 a-n, 112 may include anon-mobile device such as a desktop computer, a server, or another typeof computing device.

As mentioned above, and as shown in FIG. 1, the administrator device 112includes a survey application 114 shown thereon. In one or moreembodiments, the survey application 114 refers to a software applicationassociated with the electronic survey system 104 that facilitatesreceiving a search query, analyzing electronic survey responses, andproviding results of the analysis to the user 116 via a graphical userinterface on the administrator device 112. In one or more embodiments,the survey application 114 refers to a native application installed onthe administrator device 112. Alternatively, in one or more embodiments,the survey application 114 refers to a web browser that provides accessto information shared via a website or web-based application.Additionally, while not shown in FIG. 1, in one or more embodiments, oneor more of the respondent devices 108 a-n include a survey applicationthereon to facilitate administration of an electronic survey to theassociated respondents 110 a-n.

As an overview, the electronic survey system 104 provides an electronicsurvey to respondents 110 a-n via corresponding respondent devices 108a-n. For example, in one or more embodiments, the electronic surveysystem 104 administers an online electronic survey (e.g., via a webbrowser) including any number of electronic survey questions requestinginformation from the respondents 110 a-n. The electronic surveyquestions can include various types of questions including, for example,polls, questionnaires, censuses, or other types of questions requestinginformation from the respondents 110 a-n. In one or more embodimentsdescribed herein, the survey questions include requests for electronicsurvey responses including information contained within user-composedtext.

In one or more embodiments, the server device(s) 102 receives and storeselectronic survey responses to the administered electronic surveyquestions. In particular, the respondent devices 108 a-n can provide theelectronic survey responses to the server device(s) 102. In turn, theserver device(s) 102 stores the electronic survey responses to be madeavailable to a user 116 of the administrator device 112 upon request.For example, as will be described in further detail below, the serverdevice(s) 102 can provide electronic survey responses and/or discreteinformation contained within the electronic survey responses in responseto receiving a search query from the administrator device 112.

In one or more embodiments, the electronic survey system 104 classifiesthe electronic survey questions administered to the respondents 110 a-n.For example, the electronic survey system 104 can identify a questionclassification for an electronic survey question based on attributes ofthe electronic survey question. For instance, the electronic surveysystem 104 can classify the electronic survey question based on anidentified type of question or a requested type of information solicitedby the given electronic survey question. In turn, the electronic surveysystem 104 associates a semantics model including operators configuredto identify discrete types of information contained within correspondingelectronic survey responses. Additional detail with regard toassociating electronic survey questions with various semantics modelswill be discussed in further detail below.

As mentioned above, the electronic survey system 104 receives a searchquery received by the user 116 of the administrator device 112. Forexample, in one or more embodiments, the user 116 composes a searchquery including a request for information from a collection ofelectronic survey responses. In addition, the electronic survey system104 processes the search query by associating the search query with aquestion classification and applying an identified semantics model toelectronic survey questions associated with the question classification.In this way, the electronic survey system 104 identifies relevantelectronic survey responses and searches the electronic survey questionsto identify information as requested by the search query. Additionaldetail with regard to processing a search query and identifyinginformation from the electronic survey responses is discussed in furtherdetail below.

Upon processing the search query and analyzing the electronic surveyresponses, the electronic survey system 104 generates and provides apresentation of the results of the analysis to the user 116 of theadministrator device 112. In particular, in one or more embodiments, theelectronic survey system 104 causes the server device(s) 102 to providea presentation via the administrator device 112 by providing informationto the administrator device 112 that facilitates providing apresentation of the analysis results to the user 116 of theadministrator device 112. For example, the electronic survey system 104can cause the server device(s) 102 to provide access to information viathe network 118 to be displayed via a graphical user interface of theadministrator device 112. As another example, the electronic surveysystem 104 can cause the server device(s) 102 to provide information tothe administrator device 112 to enable the survey application 114 tolocally generate a display of the presentation to present a display ofthe presentation to the user 116. Additional detail with regard togenerating and providing the presentation to the user 116 of theadministrator device 112 as well as facilitating interactive features inconjunction with the presentation is discussed in further detail below.

FIG. 2 illustrates a flow diagram representing a process ofadministering electronic survey questions and analyzing electronicsurvey responses to the electronic survey questions. In particular, FIG.2 illustrates a process that involves receiving electronic surveyresponses including user-composed text and analyzing the user-composedtext to identify information contained therein. As shown in FIG. 2, theflow diagram includes respondent devices 108 a-n, server device(s) 102including the electronic survey system 104 and response analysis system106 thereon, and an administrator device 112 including an electronicsurvey application 114 thereon.

As illustrated in FIG. 2, the electronic survey system 104 administers202 an electronic survey question to a plurality of respondent devices108 a-n. In particular, in one or more embodiments, the electronicsurvey system 104 provides an electronic survey question including arequest for information from respondents 110 a-n associated with therespondent devices 108 a-n. In one or more embodiments, the electronicsurvey question includes a request for information in a format includinguser-composed text. The electronic survey question can include a requestfor different types of information including, for example, opinions,recommendations, questions, and/or positive or negative experiences.

In one or more embodiments, an electronic survey question includes arequest for multiple types of information. As an example, an electronicsurvey question can include a request for both a positive and negativeexperience. As another example, an electronic survey question caninclude a request for a complaint and a recommendation. As anotherexample, an electronic survey question includes a request for generalcomments, which may produce electronic survey responses includingvarious types of information.

In one or more embodiments, the electronic survey system 104 administersthe electronic survey question as part of an electronic survey includinga plurality of any number of electronic survey questions. In one or moreembodiments, the electronic survey system 104 includes questions havingdifferent formats. For example, the electronic survey system 104 mayadminister an electronic survey question including one or more questionsthat solicit a user-composed response as well as one or more questionsincluding other types of responses (e.g., numerical, binary,categorical). It will be understood that the electronic survey system104 can administer any number of electronic surveys including any numberof electronic survey questions having a variety of different formats.

As further shown in FIG. 2, the server device(s) 102 receives 204electronic survey responses. In particular, the respondent devices 108a-n provide electronic survey responses including information providedin response to the electronic survey question. In one or moreembodiments, the electronic survey responses include user-composed textincluding opinions, recommendations, questions, and other types ofinformation provided based on the electronic survey question (e.g.,based on information requested within the electronic survey question).

As shown in FIG. 2, the electronic survey system 104 classifies 206 theelectronic survey questions. In particular, prior to or afteradministration of an electronic survey, the electronic survey system 104can generate, identify, or otherwise determine a question classification(or simply “classification”) for respective electronic survey questionsof the electronic survey. In one or more embodiments, the electronicsurvey system 104 determines a classification for each electronic surveyquestion based on one or a combination of attributes of each electronicsurvey question. For example, the electronic survey system 104 cananalyze an electronic survey question using one of a variety of analysistechniques (e.g., natural language processing) to identify variousattributes of the electronic survey question and determine, based on theidentified attributes, a classification for the electronic surveyquestion.

In one or more embodiments, the electronic survey system 104 determinesa classification based on a type of the electronic survey question. Inparticular, the electronic survey system 104 can classify an electronicsurvey question based on a type of information requested within theelectronic survey question. For instance, where the electronic surveysystem 104 determines that an electronic survey question includes arequest for opinions about a product or service, the electronic surveysystem 104 identifies an “opinions about” classification and associatesthe electronic survey question with the identified classification.

In one or more embodiments, the electronic survey system 104 determinesmultiple classifications for an electronic survey question based on adetermination that the electronic survey question includes a request formultiple types of information. For example, where an electronic surveyquestion includes a request for both opinions and questions from arespondent, the electronic survey system 104 can determine aclassification (or multiple classifications) that encompasses an“opinions of” question-type and a “questions about” question-type. Othercombinations of types of information are possible as is understood basedon the disclosure herein.

In addition, or as an alternative, in one or more embodiments, theelectronic survey system 104 determines a classification based on atopic or subject of the electronic survey question. For example, theelectronic survey system 104 can determine that an electronic surveyquestion includes a question about a specific product and classify theelectronic survey question based on the specific product. Accordingly,in addition to classifying an electronic survey question based on a typeof information requested by an electronic survey question, theelectronic survey system 104 can further classify the electronic surveyquestion based on an identified topic or subject of the electronicsurvey question. In one or more embodiments, the electronic surveysystem 104 identifies the topic or subject based on an analysis (e.g.,natural language analysis) of the electronic survey question.

As further shown in FIG. 2, the electronic survey system 104 associates208 the electronic survey question with a corresponding semantics model.In particular, in one or more embodiments, the electronic survey system104 associates an electronic survey question with a semantics modelconfigured to identify information included within electronic surveyresponses to the electronic survey question. For example, in one or moreembodiments, the electronic survey system 104 identifies, generates, orotherwise determines a semantics model configured to identifyinformation included within user-composed text of any number ofelectronic survey responses. As will be described in further detailbelow in connection with FIG. 3, a semantics model includes one or moreoperators and operator functions that the electronic survey system 104can apply to user-composed text of the electronic survey responses toidentify information contained therein. In addition, the electronicsurvey system 104 can associate any number of electronic surveyquestions with corresponding semantics models and apply the multiplesemantics models to each of the electronic survey responses.

In one or more embodiments, the electronic survey system 104 associatesan electronic survey question with a semantics model based on thedetermined question classification for the electronic survey question.As an example, in one or more embodiments, the electronic survey system104 identifies a pre-existing semantics model associated with a questionclassification. In particular, because the pre-existing semantics modelincludes operators and functions configured to identify a specific typeof information requested by the electronic survey question, theelectronic survey system 104 associates the pre-existing semantics modelwith the electronic survey question and subsequently uses thepre-existing semantics model to analyze electronic survey responsesreceived in response to the electronic survey question. For instance,where the electronic survey system 104 classifies an electronic surveyquestion as “opinions of” classification (e.g., an electronic surveyquestion requesting opinions of a product or service), the electronicsurvey system 104 can identify a pre-existing semantics model includingoperators configured to identify words or phrases indicative of opinionsincluded within user-composed text of the electronic survey responses.

As an alternative to identifying and associating a pre-existingsemantics model with an electronic survey question, in one or moreembodiments, the electronic survey system 104 generates a semanticsmodel for the electronic survey question(s). For example, in one or moreembodiments, the electronic survey system 104 constructs a semanticsmodel based on the classification and/or attributes of the electronicsurvey question. In one or more embodiments, the electronic surveysystem 104 constructs the semantics model by identifying operators andoperator functions to include as part of the semantics model. Inaddition, the electronic survey system 104 can further customize agenerated semantics model or a pre-existing semantics model in a varietyof ways, as described in further detail below in connection with FIGS.3-5C.

As further shown in FIG. 2, the server device(s) 102 receive 210 asearch query from the administrator device 112. For example, in one ormore embodiments, a user 116 of the administrator device 112 composes asearch query including a request for information from a plurality ofelectronic survey responses. Alternatively, in one or more embodiments,the survey application 114 provides an interface via a display of theadministrator device 112 including selectable options to enable the user116 of the administrator device 112. Additional detail with regard togenerating a search query is provided below in connection with FIGS.5A-5C.

As shown in FIG. 2, the electronic survey system 104 classifies 212 thesearch query. In particular, in one or more embodiments, the electronicsurvey system 104 determines a query classification for the search querysimilar to one or more embodiments for determining the questionclassification for the electronic survey question described herein. Forexample, the electronic survey system 104 can classify the search queryby analyzing the search query and determining a type of informationrequested by the search query. As an example, where a search query reads“what are customer opinions about our product?,” the electronic surveysystem 104 identifies a query classification for the search query as a“questions about” classification.

In one or more embodiments, the electronic survey system 104 determinesmultiple query classifications for a search query. For example, theelectronic survey system 104 can determine multiple queryclassifications for the search query based on a determination that thesearch query includes a request for multiple types of information (e.g.,questions and opinions about a topic). As another example, theelectronic survey system 104 can determine multiple queryclassifications for the search query based on a determination that thesearch query includes a request for information about multiple topics(e.g., opinions about a product and customer service). The electronicsurvey system 104 can determine any number of query classificationsbased on determined types of information and topics associated with asearch query.

As further shown in FIG. 2, the electronic survey system 104 can furtheridentify 214 that the electronic survey question has a correspondingquestion classification as the query classification for the searchquery. In particular, in one or more embodiments, the electronic surveysystem 104 identifies that the electronic survey question includes asimilar request or otherwise corresponds to the search query based onthe determined question classification and the query classification. Forexample, the electronic survey system 104 can determine that the queryclassification and the question classification correspond based onsimilar types of information requested by the electronic survey questionand the search query. In addition, the electronic survey system 104 candetermine that the classifications match based on similar topics of thedifferent classifications.

In one or more embodiments, the electronic survey system 104 identifiesmultiple electronic survey questions corresponding to the search query.For example, where multiple electronic survey questions request asimilar type of information and have similar question classifications,the electronic survey system 104 can determine that a queryclassification corresponds to each of the similar questionclassifications for multiple electronic survey questions. Accordingly,the electronic survey system 104 can determine that electronic surveyresponses to each of the electronic survey questions are likely toinclude information therein relevant to the search query.

As further shown in FIG. 2, the electronic survey system 104 analyzes216 electronic survey responses using the associated semantics model. Inparticular, the electronic survey system 104 analyzes electronic surveyresponses to the electronic survey question by applying operators andfunctions of the semantics model to each of the electronic surveyresponses received in response a corresponding electronic surveyquestion. As mentioned above, the electronic survey system 104 canidentify a subset of electronic survey responses for one or moreelectronic surveys that are likely to include relevant information tothe search query based on the question classification(s) forcorresponding electronic survey question(s). In addition, in one or moreembodiments, the electronic survey system 104 identifies and extractsspecific information from the electronic survey responses based on thesearch query.

In one or more embodiments, the electronic survey system 104 analyzesthe electronic survey responses by identifying those electronic surveyresponses that include the relevant information. For example, in one ormore embodiments, the electronic survey system 104 filters theelectronic survey responses to generate or otherwise identify a subsetof the electronic survey responses that include information requested bythe search query and/or electronic survey responses identified byindividual operators of the semantics model(s).

In one or more embodiments, the electronic survey system 104 analyzesthe electronic survey responses by tagging specific electronic surveyresponses and/or portions of the electronic survey responses thatcorrespond to the information requested by the search query. Forexample, where a search query includes a request for opinions about aproduct, the electronic survey system 104 can tag specific words orphrases (e.g., instances of the product and associated adjectives). Inaddition, as will be described in further detail below in connectionwith FIGS. 5A-5C, the electronic survey system 104 can provide avisualization of the tagged words and phrases from the differentelectronic survey responses in connection with associated topics.

By identifying a semantics model associated with a questionclassification and identifying electronic survey responses correspondingto the electronic survey question associated with the classification,the electronic survey system 104 analyzes the electronic surveyresponses using a semantics model uniquely configured to identifyinformation included therein. For example, because the electronic surveysystem 104 determines a question classification and associated semanticsmodel for analyzing electronic survey responses in response to thereceived search query, the electronic survey system 104 reduces a numberof operations and functions performed on the electronic survey responsesto identify relevant information to the search query. In addition,because the semantics model includes operators and functions foridentifying specific types of information about specific topics, theelectronic survey system 104 additionally identifies information withhigh precision and high recall while implementing fewer operators andfunctions to identify electronic survey responses and/or informationincluded therein.

In addition, by identifying question classifications and determiningelectronic survey questions that correspond to a search query, theelectronic survey system 104 reduces the number of electronic surveyresponses analyzed by analyzing only those electronic survey responsescorresponding to electronic survey questions having a specificclassification. In this way, the electronic survey system 104 avoidsanalyzing electronic survey responses unlikely to include informationrelated or otherwise relevant to a given search query. Accordingly,rather than analyzing all electronic survey responses generally, theelectronic survey system 104 analyzes only those electronic surveyresponses likely to include relevant information of a specific typeand/or about a specific topic.

As further shown in FIG. 2, the electronic survey system 104 causes theserver device(s) to provide 218 the results of the analysis to theadministrator device 112. In particular, in one or more embodiments, theelectronic survey system 104 provides the results of the analysis via agraphical user interface of the administrator device 112. As mentionedabove, in one or more embodiments, the electronic survey system 104provides a presentation of the results by rendering the results (e.g.,providing a visualization of the results) via a remote interfaceprovided by the server device(s) 102. Alternatively, in one or moreembodiments, the electronic survey system 104 provides a presentation ofthe results by providing information to the administrator device 112 andcausing the survey application 114 to provide a presentation via adisplay of the administrator device 112.

Additional detail will now be given with respect to questionclassifications and associated semantics models. In particular, FIG. 3includes a block diagram including exemplary question classifications302 a-d and associated semantics models 304 a-d. As shown in FIG. 3, afirst question classification 302 a includes an associated firstsemantics model 304 a including operators 306 a and terminology data 308a. As further shown, a second question classification 302 b includes anassociated second semantics model 304 b including an operator 306 b andterminology data 308 b. As further shown, a third questionclassification 302 c includes a third associated semantics model 304 cincluding operators 306 c and terminology data 308 c. In addition, FIG.3 shows a generic question classification 302 d and an associatedgeneric semantics model 304 d including operators 306 d. Each of thequestion classifications 302 a-d and semantics models 304 a-d caninclude similar features and functionality. Accordingly, one or morefeatures described in connection with a specific question classificationor semantics model can similarly apply to the other questionclassifications and semantics models.

As a first example shown in FIG. 3, the electronic survey system 104 canidentify and associate the first question classification 302 a with acorresponding electronic survey question. The electronic survey system104 can subsequently analyze electronic survey responses (e.g., inresponse to receiving a search query) using the first semantics model304 a. In particular, where the electronic survey system 104 associatesan electronic survey question with the first question classification 302a, the electronic survey system 104 can analyze electronic surveyresponses to the electronic survey question by applying the operators306 a (Operator A and Operator B) to the electronic survey responses toidentify discrete information corresponding to the respective operator306 a.

In one or more embodiments, the electronic survey system 104 configuresthe operators 306 a of the first semantics model 304 a to identifydifferent types of information. For example, operator A may include oneor more operator functions configured to identify opinions about anassociated topic. On the other hand, Operator B may include one or moreoperator functions configured to identify general comments about theassociated topic.

In addition, or as an alternative, in one or more embodiments, theelectronic survey system 104 configures the operators 306 a to identifysimilar types of information (or different types of information) usingdifferent analysis techniques. For example, operator A may include oneor more operator functions that identify discrete information fromuser-composed text using a pattern-matching analysis (e.g., wordmatching, phrase matching). On the other hand, Operator B may includeone or more operator functions that identify discrete information fromuser-composed text using a dependency parsing technique, contingencymatching technique, or other parsing technique for identifyinginformation from user-composed text. Accordingly, some or all of thesemantics models 304 a-d can include operators that, when applied toelectronic survey responses, identify different types of informationusing different analysis techniques.

In addition to the operators 306 a, the electronic survey system 104 cananalyze electronic survey responses in view of the terminology data 308a included as part of the first semantics model 304 a. In one or moreembodiments, the terminology data 308 a includes one or more words,phrases, and other terminology information including a specific meaningor significance with respect to a topic or context of the electronicsurvey question. For example, the terminology data 308 a can includespecific definitions for words, phrases, or other terminology. Inaddition, the terminology data 308 a can include a significance,sentiment, value, or other metric associated with words, phrases, andother terminology. In this way, the electronic survey system 104 canconsider terms of art and/or words that have unconventional meaningswhen used in relation to an associated topic, or other considerationwhen identifying information included within electronic surveyresponses.

As another example shown in FIG. 3, the electronic survey system 104 candetermine the second question classification 302 b associated with thesecond semantics model 304 b with a corresponding electronic surveyquestion. The electronic survey system 104 can analyze any electronicsurvey responses received in response to the corresponding electronicsurvey question by applying the operator 306 b (Operator C) to theelectronic survey responses in view of the terminology data 308 b of thesecond semantics model 304 b. As shown in FIG. 3, the second semanticsmodel 304 b can include unique operators 306 b and terminology data 308b from the operators 306 a and terminology data 308 a of the firstsemantics model 304 a.

As another example shown in FIG. 3, the electronic survey system 104 candetermine the third question classification 302 c associated with thethird semantics model 304 c with a corresponding electronic surveyquestion. The electronic survey system 104 can analyze any electronicsurvey responses received in response to the corresponding electronicsurvey question by applying the operators 306 c (Operator A, Operator B,Operator D) to the electronic survey responses in view of theterminology data 308 c of the third semantics model 304 c. As shown inFIG. 3, the third semantics model 304 c includes two similar operators(Operator A, Operator B) to the first semantics model 304 a in additionto a unique operator (Operator D) not included within the firstsemantics model 304 a.

In one or more embodiments, the terminology data 308 c of the thirdsemantics model 304 c can include similar or different data from theterminology data 308 a of the first semantics model 304 a. For instance,if the electronic survey questions corresponding to the first questionclassification 302 a and third question classification 302 c includesimilar topics, the terminology data 308 c of the third semantics model304 c can include similar data as the terminology data 308 a of thefirst semantics model 304 a.

As another example shown in FIG. 3, where the electronic survey system104 fails to identify a question classification (e.g., where a questionclassification or pre-defined semantics model does not exist for aparticular type of question or topic), the electronic survey system 104can identify or otherwise determine a generic question classification302 d (e.g., a default question classification) corresponding to anelectronic survey question and analyze electronic survey responses tothe electronic survey question using the generic semantics model 304 dincluding operators 306 d (Operator X, Operator Y, Operator Z). In oneor more embodiments, the operators 306 d of the generic semantics model304 d includes operators for identifying general information commonlyincluded within electronic survey responses. For example, the operators306 d can include multiple operators that use different text analysistechniques for identifying relationship data between topics and genericdescriptors. In addition, because the generic semantics model 304 drelates generally to a variety of different topics, the genericsemantics model 304 d may omit or otherwise not include terminology datasimilar to other semantics models 304 a-c associated with specifictopics.

As mentioned above, the electronic survey system 104 can associatemultiple classifications with an electronic survey question. Forexample, where an electronic survey question includes a request formultiple types of information, the electronic survey system 104 canassociate a single electronic survey question with multiple questionclassifications, which further associates the electronic survey questionwith different semantics models. In this case, the electronic surveysystem 104 can apply each of the semantics models individually.Alternatively, the electronic survey system 104 can generate a newsemantics model including each of the operators and a combination of theterminology data. In one or more embodiments, the electronic surveysystem 104 can apply each of the unique operators once without applyingduplicate operators between semantics models. For example, where twosemantics models each included Operator A, the electronic survey system104 would apply Operator A only once between the two semantics modelsrather than applying Operator A for each separate application of thesemantics models.

Additional detail will now be given with respect to an example operatorin accordance with one or more embodiments described herein. Inparticular, FIG. 4 includes a pattern matching operator 402 foranalyzing an electronic survey response associated with a givensemantics model that includes the example pattern matching operator 402.While FIG. 4 illustrates an example operator configured to identifypatterns of words and phrases within an electronic survey response, itwill be understood that operators can identify content included withinelectronic survey responses in a variety of ways.

For example, while FIG. 4 illustrates an example pattern matchingoperator 402 that identifies patterns of text and phrases withinelectronic survey responses, in one or more embodiments, the electronicsurvey system 104 can implement operators that utilize differentanalysis techniques. Examples of analysis techniques that the electronicsurvey system 104 can incorporate include distance-based entities-aspectpairs, meronym discriminators to filter out unlikely entities andaspects, term frequency-inverse document frequency (TF-IDF), rankingentities and aspects of text and phrases, pattern-based filters,negation detection, part-of-speech regular expressions (regexes),association-rule mining, C-value measure, information distance andword-similarity, co-reference resolution, double propagation andvariants thereof.

In one or more embodiments, the electronic survey system 104 generatesor otherwise identifies semantics models including a variety ofoperators for analyzing user-composed text. For example, the semanticsmodels can include operators that use one or more text analysistechniques described in “Mining and Summarizing Customer Reviews” by Huand Lu, “Extracting Product Features and Opinions From Reviews” byPopescu and Etzioni, “Opinion Extraction, Summarization, Tracking inNews and Blog Corpora” by Ku et al., “Red-Opal: Product Feature Scoringfrom Reviews,” by Scaffidi et al., “An Unsupervised Opinion Miner fromUnstructured Product Reviews,” by Moghaddam and Ester, “Multi-AspectOpinion Polling from Textual Reviews,” by Zhu et al., “A ReviewSelection Approach for Accurate Feature Rating Estimation,” by Long etal., and “Opinion Mining With Deep Recurrent Neural Networks,” by Irsoyet al. Each of these publications are incorporated herein by referencein its entirety. In addition, or as an alternative, the electronicsurvey system 104 generates or otherwise identifies semantics modelsincluding operators for analyzing user-composed text using one or moretechniques described in “The Second Workshop on Natural LanguageProcessing for Social Media in conjunction with COLING-2014,” which isalso incorporated herein by reference in its entirety.

For the sake of explanation, FIG. 4 illustrates an example operatorincluding operator functions in accordance with one or more embodiments.In particular, FIG. 4 illustrates an example pattern matching operator402 for analyzing an electronic survey response and identifying opinionsof a topic therein that utilizes a pattern-matching technique toidentify patterns of words to identify one or more opinions statedwithin a sample electronic survey response. For example, as shown inFIG. 4, the electronic survey response reads “The lightweight tent wassurprisingly durable and cheap.” As shown in FIG. 4, the electronicsurvey system 104 initiates the pattern matching operator 402 bybreaking down the electronic survey response. In particular, as shown inFIG. 4, the electronic survey system 104 breaks down the electronicsurvey response word-by-word and performs a lemmatization of theelectronic survey response. The electronic survey system 104 furtherbreaks down the lemmatization of the electronic survey response togenerate a parts of speech representation of the electronic surveyresponse. Accordingly, as shown in FIG. 4, the electronic survey system104 breaks down “The lightweight tent was surprisingly durable andcheap” to read as a sequence of parts of speech including “DEF, ADJ,NOUN, VERB, ADV, ADJ, CONJ, ADJ” where DEF refers to a definite article,ADJ refers to an adjective, NOUN refers to a noun, VERB refers to averb, ADV refers to an adverb, and CONJ refers to a conjunction.

As further shown in FIG. 4, the electronic survey system 104 identifiesa topic 404 of the electronic survey response. For example, as shown inFIG. 4, the pattern matching operator 402 includes a topic defined by anoun [N]. Thus, where the electronic survey system 104 identifies a nounof “tent” within the electronic survey response, the electronic surveysystem 104 determines that the topic 404 of the electronic survey is atent. In one or more embodiments, the electronic survey system 104identifies multiple topics. Alternatively, in one or more embodiments,the electronic survey system 104 identifies multiple phrases orsentences within the electronic survey response and identifies one topicper phrase or sentence. Accordingly, in one or more embodiments, theelectronic survey system 104 applies an operator to multiple phrases orindividual sentences that make up the electronic survey response.

In one or more embodiments, the electronic survey system 104 performssimilar steps of breaking down the electronic survey response (e.g.,performing the literal breakdown, lemmatization, and parts of speechbreakdown). In addition, because many operators (e.g., operators of thesame semantics model) include these similar steps, in one or moreembodiments, the electronic survey system 104 breaks down the electronicsurvey response and identifies the topic of the electronic surveyresponse for each of the operators for a given semantics model.Accordingly, where a semantics model includes multiple operators, in oneor more embodiments, rather than breaking down the electronic surveyresponse for each operator, the electronic survey system 104 breaks downthe electronic survey response, identifies a topic, and/or otherpreliminary analysis steps and applies the respective operators to thebroken-down text of the electronic survey response.

As shown in FIG. 4, the pattern matching operator 402 includes aplurality of operator functions 406 a-f for identifying discreteinstances of information contained within the electronic surveyresponse. In particular, each of the operator functions 406 a-f areconfigured to identify a corresponding pattern of words (e.g., patternsof parts of speech) within the electronic survey response. As shown inFIG. 4, the electronic survey system 104 applies each of the operatorfunctions 406 a-f and determines whether the electronic survey responseincludes a phrase or pattern of words that correspond to a patterndefined by the respective operator functions 406 a-f.

For example, as shown in FIG. 4, the pattern matching operator 402includes a first operator function 406 a defined as “<ADJ>[N].” As shownin FIG. 4, the electronic survey system 104 identifies “lightweighttent” based on the broken down parts of speech matching the patternidentified by the first operator function 406 a. Based on thisidentification, the electronic survey system 104 can further tag orotherwise identify “lightweight” as an opinion of the tent includedwithin the electronic survey response.

The pattern matching operator 402 further includes a second operatorfunction 406 b defined as “<ADJ><ADJ><N>,” in which the electronicsurvey system 104 attempts to identify a corresponding pattern withinthe parts of speech breakdown of the electronic survey response. Asshown in FIG. 4, the electronic survey system 104 fails to identify anymatches within the parts of speech breakdown for the electronic surveyresponse. As a result, the electronic survey system 104 takes noadditional action with respect to the second operator function 406 b andcontinues to apply other operator functions of the pattern matchingoperator 402.

The pattern matching operator 402 additionally includes a third operatorfunction 406 c defined as “[N][Lem:Be]<ADJ>,” in which the electronicsurvey system 104 attempts to identify a corresponding pattern withinthe parts of speech breakdown and lemma breakdown of the electronicsurvey response. As shown in FIG. 4, the electronic survey system 104fails to identify any matches within the parts of speech or lemmabreakdowns of the electronic survey response. As a result, theelectronic survey system 104 takes no additional action with respect tothe third operator function 406 c and continues to apply other operatorfunctions of the pattern matching operator 402.

The pattern matching operator 402 further includes a fourth operatorfunction 406 d defined as “[N][Lem:Be]<ADV><ADJ>.” As shown in FIG. 4,the electronic survey system 104 identifies “tent was surprisinglydurable” based on the broken down parts of speech matching the patternidentified by the fourth operator function 406 d. Based on thisidentification, the electronic survey system 104 can tag or otherwiseidentify “durable” as another opinion included within the electronicsurvey response.

The pattern matching operator 402 additionally includes a fifth operatorfunction 410 e defined as “[Fn 3]<CONJ><ADJ>,” which includes the samebreakdown of the third operator function 410 c as well as a conjunctionand adjective. The electronic survey system 104 attempts to identify thecorresponding pattern within the parts of speech breakdown and lemmabreakdown of the electronic survey response. Because the third operatorfunction 410 c included no matches within the electronic surveyresponse, the electronic survey system 104 can skip applying the fifthoperator function 410 e and return a result indicating that no matchesexist within the electronic survey response for the fifth operatorfunction 410 e and continue to apply other operator functions of thepattern matching operator 402 to the electronic survey response.

The pattern matching operator 402 additionally includes a sixth operatorfunction 410 f defined as “[Fn 4]<CONJ><ADJ>,” which includes the samebreakdown of the fourth operator function 410 d as well as a conjunctionand adjective. Similar to the other operator functions, the electronicsurvey system 104 attempts to identify the corresponding pattern withinthe parts of speech breakdown and lemma breakdown of the electronicsurvey response. Because the fourth operator function 410 d returned apositive match, the electronic survey system 104 can additionally searchfor and identify the additional parts of the pattern included within thesixth operator function 410 f. In particular, as shown in FIG. 4, theelectronic survey system 104 identifies “tent was surprisingly durableand cheap.” Based on this identification, the electronic survey system104 can tag or otherwise identify “cheap” as another instance of anopinion included within the electronic survey response.

It will be understood that while the pattern matching operator 402 onlyincludes six operator functions 410-a-f, operators that make up acorresponding semantics model can include any number of operatorfunctions to identify discrete information included within an electronicsurvey response. In addition, in one or more embodiments, the electronicsurvey system 104 adds additional operators and operator functions aswell as modifies existing operators and operator functions over time.

For example, in one or more embodiments, the electronic survey system104 trains a machine learning model that generates a semantics modelincluding operators and operator functions based training data includingelectronic survey responses and identified opinions included therein. Inparticular, the electronic survey system 104 can analyze the electronicsurvey responses and identified opinions to generate one or moresemantics models that accurately identify opinions within the electronicsurvey responses of the training data. The electronic survey system 104can utilize the semantics model generated by the machine learningprocess to analyze subsequently received electronic survey responses. Asadditional information becomes available, the electronic survey system104 can additionally update the semantics model to include opinions andopinion operators that more accurately identify opinions included withinelectronic survey responses.

In addition, with respect to identified opinions, experiences, or otherinformation that includes a value of user (e.g., respondent) sentiment(e.g., opinions about a topic), the electronic survey system 104 canfurther associate a sentiment value with the identified information. Forexample, the electronic survey system 104 can identify a generalpositive or negative sentiment for an electronic survey response basedon general meanings of identified words and phrases therein. In one ormore embodiments, the electronic survey system 104 identifies a positiveor negative sentiment for an electronic survey response based on ananalysis of the electronic survey response as a whole. For example,where the meaning of a specific word is neutral or unknown, theelectronic survey system 104 can nonetheless associate a positive ornegative sentiment with the word based on a context in which the word isused. For instance, where a neutral or ambiguous word is used in contextwith other positive opinions, the electronic survey system 104 maydetermine that the word has a positive meaning within the context of theelectronic survey response.

As an example, in connection with the electronic survey response shownin FIG. 4, while the adjective “cheap” can include either a positive ornegative meaning, the electronic survey system 104 may nonethelessassociate “cheap” for the example electronic survey response with apositive sentiment based on other positive sentiments expressed withinthe electronic survey response. Nevertheless, where other electronicsurvey responses include “cheap” used in connection with negativeadjectives (e.g., “the tent was cheap and flimsy”), the electronicsurvey system 104 may ultimately determine that “cheap” has an overallnegative meaning. In one or more embodiments, the electronic surveysystem 104 may determine that “cheap” has an ambiguous meaning based onmixed or conflicting uses of the term across the electronic surveyresponses.

In one or more embodiments, the electronic survey system 104 assigns orassociates a positive or negative sentiment for a word, phrase, orelectronic survey response based on terminology data included within thesemantics model. For example, where a word (e.g., an adjective) thatdescribes a topic has a specific meaning different or in contrast to ageneral meaning when used in connection with the topic, the electronicsurvey system 104 can include the specific meaning or sentiment withinthe terminology data for a corresponding semantics model. In one or moreembodiments, a user 116 of the administrator device 112 can modify orotherwise re-classify a term for a semantics model, as will be describedin further detail in connection with FIGS. 5A-5C.

As mentioned above, in addition to analyzing the electronic surveyresults using operators of one or more semantics models, the electronicsurvey system 104 can further provide a presentation of the results ofthe analysis of electronic survey responses to a user 116 of anadministrator device 112. For example, in one or more embodiments, theelectronic survey system 104 provides the results of the analysis of acollection of electronic survey results via a graphical user interfaceof the administrator device 112. Additional detail will now be givenwith respect to an example graphical user interface illustrated on aclient device shown in FIGS. 5A-5C. In addition, while one or moreembodiments described herein relate to the electronic survey system 104providing features and characteristics of a presentation of analysisresults

For example, FIGS. 5A-5C illustrate a client device 502 including agraphical user interface 504 for displaying a presentation of results ofan analysis of a plurality of electronic survey responses. Inparticular, the client device 502 described in connection with FIGS.5A-5C shows an example administrator device 112 having a surveyapplication 114 thereon in accordance with one or more embodiments. Inaddition, while FIGS. 5A-5C show a client device 502 including apersonal computer device, the electronic survey system 104 can similarlyprovide a presentation of the results of the analysis of electronicsurvey responses via a mobile device or other computing device.

As shown in FIG. 5A, the graphical user interface 504 includes a sidebarmenu 506 including selectable options. In one or more embodiments, theelectronic survey system 104 provides a presentation of selectiveresults of analyzing a collection of electronic survey responses basedon a selected option from the sidebar menu 506 of selectable options.For example, the electronic survey system 104 can provide a display ofelectronic survey results organized by topic, types of responses (e.g.,recommendations, co-occurrences, opinions), and various themes. In theexample shown in FIG. 5A, the electronic survey system 104 provides apresentation of opinions identified within electronic survey resultsbased on detecting a user selection of the opinions option.

As further shown in FIG. 5A, the graphical user interface 504 includesone or more graphical elements that enable a user of the client device502 to compose or otherwise identify a search query. For example, thegraphical user interface 504 includes a free-form query interface 508including a space wherein a user of the client device 502 can enter atext query. Similar to one or more embodiments described herein, theelectronic survey system 104 can analyze the text query to identify orotherwise determine a query classification. Based on the queryclassification, the electronic survey system 104 can identify any numberof electronic survey questions having a question classificationcorresponding to the classification of the search query.

In one or more embodiments, and as shown in FIG. 5A, the graphical userinterface 504 further includes a search icon query interface 510including selectable icons that facilitate creation of a search query.In particular, a user of the client device 502 can select icons toidentify one or more topics and types of information included within thequery. In one or more embodiments, the search icon query interface 510includes different selectable options based on the selected option ofthe sidebar menu 506. Accordingly, because the opinions option has beenselected, the electronic survey system 104 provides selectable icons forgenerating a search query for identifying opinions of respondentsincluded within a collection of electronic survey responses.

As further shown in FIG. 5A, the graphical user interface 504 includes adisplay of identified responses 512 that include informationcorresponding to the search query. For example, as shown in FIG. 5A, thesearch query includes an identification of a topic of “tents” and arequest for positive opinions about the tents. In response to receivingthis search query, the electronic survey system 104 identifieselectronic survey results including an identified topic of tents andidentified positive opinions. In accordance with one or more embodimentsdescribed herein, the electronic survey system 104 identifies theelectronic survey results based on applying a semantics model toelectronic survey results associated with electronic survey questionsincluding a request for opinions related to the tent (e.g., based on aquestion classification for the electronic survey question(s)). Inparticular, the display of identified responses 512 includes thoseelectronic survey responses including a mention of a tent or tent inaddition to one or more adjectives associated with the tent or tents.

FIG. 5B illustrates an example presentation including a display ofresults of performing an analysis of electronic survey responses basedon a received search query. In particular, as shown in FIG. 5B, thegraphical user interface 504 includes a text-based search query 514including user-composed text and a visualization 516 of identified termsfrom within electronic survey results corresponding to the search query.

For example, in response to receiving the search query “What are peoplesaying about our tents?,” the electronic survey system 104 can determinea query classification or otherwise classify the search query as a“questions about” search query having a topic of “tents.” In addition,the electronic survey system 104 can identify one or more electronicsurvey questions having a similar question classification (e.g.,opinions about tents or other products associated with a respectivemerchant or industry). The electronic survey system 104 can furtheranalyze electronic survey results corresponding to the identified one ormore electronic search questions and track words or phrases associatedwith opinions about tents using one or more semantics models associatedwith the identified electronic survey questions having one or morequestion classification(s) associated with the query classification.

As shown in FIG. 5B, the electronic survey system 104 provides avisualization 516 of the identified opinions about the tents. Inparticular, the visualization 516 includes instances of positiveopinions including words identified within electronic survey responseslike good, big, roomy, durable, sturdy, user-friendly, easy to use,awesome, and favorite. The visualization 516 additionally includesinstances of negative opinions including words and phrases identifiedwithin electronic survey responses like awful, small, weak, impossible,and confusing. In one or more embodiments, the electronic survey system104 provides identified words and phrases having a threshold number ofinstances that appear within the electronic survey responses. As anotherexample, the electronic survey system 104 can provide identified wordsand phrases that appear most often in the respective categories.

In one or more embodiments, the electronic survey system 104 can selectan option to “see more” positive or negative opinions. For example,where the electronic survey system 104 only provides a portion of theidentified words and phrases identified a threshold number of times, theelectronic survey system 104 can provide other words or phrases thatexpress positive or negative opinions that appear fewer than thethreshold number of times. In one or more embodiments, the electronicsurvey system 104 facilitates switching between the visualization shownin FIG. 5B and the listing electronic survey results shown in FIG. 5A.Alternatively, in one or more embodiments, the electronic survey system104 provides both the visualization 516 and a listing of identified oneor more electronic survey results within the graphical user interface504.

In some embodiments, the words and phrases shown within thevisualization 516 include selectable icons that enable a user of theclient device 502 to view specific electronic survey responses includingspecific words or phrases. For example, in response to detecting a userselection of “awful” under the negative opinions category, theelectronic survey system 104 can provide a listing of electronic surveyresults including instances of the word “awful” used in connection withthe tents within relevant electronic survey responses.

As further shown in FIG. 5B, the visualization 516 of the analysisresults includes neutral and/or unknown terms. For example, thevisualization 516 includes neutral words and phrases like “not bad” and“OK.” In addition, the visualization 516 includes unknown words like“cheap.” In one or more embodiments, the electronic survey system 104categorizes the identified words and phrases based on conventionalmeanings of the terms. Alternatively, in one or more embodiments, theelectronic survey system 104 categorizes the identified words andphrases based on terminology data included within semantics models usedto analyze the electronic survey responses. For example, where “big” or“small” could be good or bad depending on the topic, the electronicsurvey system 104 can determine that “big” is positive and that “small”is negative based on terminology data of a semantics model includingassigned sentiment to various terms.

As mentioned above, in one or more embodiments, the electronic surveysystem 104 determines a sentiment associated with one or more words orphrases based on an analysis of the electronic survey responses in whichthe words or phrases appear. For example, where a particular descriptor(e.g., word or phrase) appears in connection with other positive terms,the electronic survey system 104 can associate the descriptor with apositive sentiment. In addition, where a descriptor appears inconnection with both negative and positive terms, the electronic surveysystem 104 can categorize the descriptor as unknown or ambiguous. Forexample, where the word “cheap” may appear both in connection withpositive and negative terms, the electronic survey system 104 can assign“cheap” to an unknown category.

In one or more embodiments, the electronic survey system 104 enables auser of the client device 502 to re-categorize or manually assign a wordor phrase to a category. For example, FIG. 5C shows a “cheap” word icon518 re-assigned from the unknown category to the positive opinionscategory. In particular, the electronic survey system 104 changes theword “cheap” from the unknown category to the positive opinion categorybased on detecting a user selection of the word icon 518 and movement ofthe word icon 518 from the unknown category to the positive opinioncategory.

In one or more embodiments, the electronic survey system 104 modifies asemantics model based on the user input moving a word or phrase from onecategory to another category. For example, in response to detectingplacement of the icon 518 from the unknown category to the positiveopinions category, the electronic survey system 104 can modifyterminology data for a semantics model used to analyze the electronicsurvey responses including the information shown within thevisualization 516 of the survey results analysis.

In addition, as mentioned above, in one or more embodiments, theelectronic survey system 104 provides a listing of one or moreelectronic survey results including the words or phrases identifiedtherein. In one or more embodiments, the electronic survey system 104enables a user of the client device 502 to select one or more words orphrases where particular words or phrases were missed in analyzing theelectronic survey results. In one or more embodiments, the electronicsurvey system 104 further modifies a semantics model based on anidentification of one or more words or phrases within an electronicsurvey response. For example, the electronic survey system 104 can addor modify one or more operator functions to avoid future instances ofmissing relevant words or phrases. Accordingly, the electronic surveysystem 104 can modify semantics models, operators, or individualoperator functions over time and based on received user input to improveupon the accuracy of identifying information contained within electronicsurvey responses.

FIG. 6 illustrates a schematic diagram of an example embodiment of theserver device(s) 102 including the electronic survey system 104 and theresponse system 106 described above in connection with FIGS. 1-5C. Asfurther shown in FIG. 6, the response analysis system 106 includes asurvey administrator 602, question classifier 604, semantics modelmanager 606, which includes an operator manager 608 and a terminologymanager 610, and a presentation manager 612. The response analysissystem 106 further includes a data storage 614 including survey data 616and semantics data 618. Although FIG. 6 illustrates the components104-106 and 602-614 to be separate, any of the components 104-106 or602-614 may be combined into fewer components, such as into a singlefacility or module or divided into more components as may be suitablefor one or more embodiments. In addition, the components 104-106 or602-614 may be located on or implemented by one or more computingdevices, such as those described below.

In addition, components 104-106 and 602-614 can comprise software orhardware or a combination of software and hardware. For example, thecomponents 104-106 and 602-614 can comprise one or more instructionsstored on a non-transitory computer readable storage medium that areexecutable by a processor of one or more computing devices. Whenexecuted by the one or more processors, the computer-executableinstructions of the electronic survey system 104 cause computingdevice(s) to perform the methods described herein. Alternatively, thecomponents 104-106 and 602-614 can comprise a special-purpose processingdevice to perform a certain function. Additionally or alternatively, thecomponents 104-106 and 602-614 can comprise a combination ofcomputer-executable instructions and hardware.

As mentioned above, the response analysis system 106 includes a surveyadministrator 602. The survey administrator 602 collects or otherwisereceives electronic survey information from respondents 110 a-n of oneor more electronic surveys. For example, in one or more embodiments, thesurvey administrator 602 administers an electronic survey by providingelectronic survey questions to respondent devices 108 a-n. In one ormore embodiments, the survey administrator 602 receives and storeselectronic survey responses received from the respondent devices 108 a-nin response to the electronic survey questions. As discussed above, thesurvey administrator 602 receive any number of electronic surveyresponses to any number of electronic survey questions. In addition, thesurvey administrator 602 can receive electronic survey responsesincluding user-composed text that includes information composed byrespondents 110 a-n of the respondent devices 108 a-n.

As shown in FIG. 6, the response analysis system 106 further includes aquestion classifier 604. The question classifier 604 classifies one ormore electronic survey questions based on a variety of attributes of theelectronic survey question(s). For example, the question classifier 604can classify an electronic survey question as a question-type referringto a type of information requested by the electronic survey question. Inaddition, or as an alternative, the question classifier 604 can classifythe electronic survey question based on a topic (or multiple topics) ofthe electronic survey question.

As further shown in FIG. 6, the response analysis system 106 includes asemantics model manager 606. In one or more embodiments, the semanticsmodel manager 606 determines, identifies, or otherwise associates theidentified question classification with a corresponding semantics modelfor analyzing electronic survey responses received in response to theelectronic survey question associated with the question classification.In particular, the semantics model manager 606 determines a semanticsmodel to use for analyzing any electronic survey responses received inresponse to one or more electronic survey questions associated with thequestion classification. In one or more embodiments, the semantics modelmanager 606 identifies a semantics model for analyzing a subset of acollection of electronic survey responses (e.g., a collection ofelectronic survey responses corresponding to all electronic surveyresponses of an electronic survey).

As shown in FIG. 6, the semantics model manager 606 includes anoperation manager 608. As mentioned above, a semantics model includesone or more operators for identifying discrete information containedwithin electronic survey responses. In one or more embodiments, theoperation manager 608 generates or otherwise identifies operators toinclude within a semantics model. For example, the operation manager 608can selectively identify operators including operator functions toidentify specific patterns, dependencies, or other content of theelectronic survey responses to identify information included withinuser-composed text of the electronic survey responses.

In one or more embodiments, the operation manager 608 trains or modifiesthe operators and/or operator functions of a semantics mode. Forexample, in one or more embodiments, the operation manager 608 receivesa set of training data including electronic survey responses in whichinformation has been identified. The operation manager 608 trains thesemantics model by generating or selecting operators and operatorfunctions configured to identify information within the electronicsurvey questions with a threshold accuracy (e.g., a threshold recallrate and/or a threshold precision rate). In one or more embodiments, asadditional information comes available, the operation manager 608 canrefine the semantics model by adding or modifying operators therein toimprove upon the rate of recall and precision in identifying informationcontained within subsequently received electronic survey responses.

As described in one or more embodiments herein, operation manager 608identifies and implements operators including patterns, dependencies,etc. to identify discrete information within user-composed text ofelectronic survey responses. In addition, or as an alternative, in oneor more embodiments, operation manager 608 trains or otherwiseimplements a semantics model that identifies neural opinion operatorsthat use bi-directional long-term short memory (LTSM) with an attentionmechanism to extract or otherwise identify opinion-subject pairs fromarbitrary text. In one or more embodiments, operation manager 608 canutilize other neural network architectures with labeled data (e.g.,LSTM) to learn patterns not explicitly specified, but can be inferredfrom labeled examples.

In addition to the operation manager 608, the semantics model manager606 further includes a terminology manager 610. In one or moreembodiments, the terminology manager 610 identifies words and phrasesassociated with various topics and generates terminology data forcorresponding topics and semantics models. In addition, the terminologymanager 610 can associate meanings with words and phrases for use inidentifying information included within electronic survey responses. Forexample, when identifying positive and/or negative opinions about atopic, the terminology manager 610 can identify specific meanings orsentiment associated with different words or phrases in context with thetopic that have a different meaning or sentiment when used in othercontexts. In one or more embodiments, the terminology manager 610maintains terminology data for each respective semantics model and/orfor each topic of multiple topics.

As further shown in FIG. 6, the response analysis system 106 includes apresentation manager 612. In one or more embodiments, the presentationmanager 612 generates and provides a presentation of results of ananalysis of electronic survey responses to be displayed via a graphicaluser interface of an administrator device 112. In one or moreembodiments, the presentation manager 612 provides the presentation viaa web interface or other interface provided via a remote source to theadministrator device 112. Alternatively, in one or more embodiments, thepresentation manager 612 provides data to the administrative device 112which enables a survey application 114 to generate and display thepresentation via a graphical user interface on the administrative device112.

Similar to one or more embodiments described herein, the presentationmanager 612 further provides interactive features that enable a user 116of an administrator device 112 to view different information associatedwith electronic survey responses. For example, the presentation manager612 can provide a query interface that enables a user to enter a searchquery or select various options for generating a search query. Inaddition, the presentation manager 612 provides selectable options thatenable the user 116 to view specific responses that include discreteinformation. Further, the presentation manager 612 provides selectableoptions that facilitates re-categorization of various words or phrases(e.g., re-assigning meaning or sentiment to particular words or phrasesfor a topic). In one or more embodiments, the presentation manager 612facilitates modification of one or more semantics models based onreceived input with respect to a presentation of results of analyzingthe electronic survey responses.

As further shown in FIG. 6, the electronic survey system 104 includes adata storage 614 including survey data 616. In one or more embodiments,the survey data 616 includes any data associated with one or moreelectronic surveys administered by the electronic survey system 104. Forexample, the survey data 616 can include information associated withelectronic survey questions as well as information included withinelectronic survey responses.

In addition, the data storage 614 includes semantics data 618. Thesemantics data 618 includes any information associated with semanticsmodels used for analyzing electronic survey results. For example, thesemantics data 618 can include stored associations between questionclassifications and respective semantics models. In addition, thesemantics data 618 can include information associated with any number ofoperators as well as individual operator functions that make up thevarious operators.

FIGS. 1-6, the corresponding text, and example, provide a number ofdifferent systems and devices that facilitate analyzing electronicsurvey responses and providing a presentation of the analysis results.In addition to the foregoing, embodiments can also be described in termsof flowcharts including acts and steps in a method for accomplishing aparticular result. For example, FIGS. 7-8 illustrate flowcharts ofexemplary methods and acts in accordance with one or more embodiments.

FIGS. 7-8 illustrates a flowchart of one example method 700 of analyzingelectronic survey responses to one or more electronic survey questionsand providing a presentation of analysis results. While FIGS. 7-8illustrate example steps according to one or more embodiments, otherembodiments may omit, add to, reorder, and/or modify any of the stepsshown in FIGS. 7-8. Additionally, one or more steps shown in FIGS. 7-8may be performed by a client device, server device, or combination ofcomponents located thereon.

As shown in FIG. 7, a method 700 includes an act 710 of receivingelectronic survey responses to an electronic survey question. Forexample, in one or more embodiments, the act 710 includes receiving,from a plurality of respondent devices 108 a-n, a plurality ofelectronic survey responses to an electronic survey question where theplurality of electronic survey responses including user-composed text.In one or more embodiments, receiving electronic survey responsesinvolves receiving electronic survey responses from any number ofelectronic survey questions (from one or multiple electronic surveys).

As further shown in FIG. 7, the method 700 includes an act 720 ofdetermining a question classification associated with a semantics modelfor the electronic survey question. For example, in one or moreembodiments, the act 720 includes determining, for the electronic surveyquestion, a question classification associated with a semantics modelwhere the semantics model includes one or more operators for analyzingelectronic survey responses to the electronic survey question. In one ormore embodiments, the method 700 includes identifying the semanticsmodel associated with the question classification by identifying, from aplurality of pre-defined semantics models, a first pre-defined semanticsmodel associated with a type of the electronic survey question. In oneor more embodiments, identifying the question classification includesfailing to identify a predefined question classification for theelectronic survey question and, in response to failing to identify thepredefined question classification for the electronic survey question,the method 700 includes identifying a default question classificationassociated with a default semantics model comprising multiple operatorsfor analyzing user-composed text.

In one or more embodiments, the method 700 includes generating asemantics model by identifying the one or more operators configured toidentify one or more types of information contained within user-composedtext. The different types of information can include one or more ofopinions, recommendations, questions, or other types of informationcontained within electronic survey responses. In one or moreembodiments, the semantics model includes a predetermined set of the oneor more operators corresponding to a type of the electronic surveyquestion. Alternatively, in one or more embodiments, the semantics modelincludes any number of individually selected operators configured toidentify information within user-composed text.

In one or more embodiments, the operators of the semantics model includeterminology data including definitions of one or more words or phrasesassociated with a topic of the electronic survey question. For example,the terminology data can include identified words, phrases, andparticular definitions different from conventional meanings for thewords or phrases in other contexts. In one or more embodiments, the oneor more operators include a plurality of respective operator functions.As an example, in one or more embodiments, an operator includes operatorfunctions where each operator function includes a pattern of words orphrases to search within the user-composed text for each of theplurality of electronic survey responses.

As further shown in FIG. 7, the method 700 includes an act 730 ofreceiving a search query requesting information associated with theplurality of electronic survey responses. As further shown, the method700 includes an act 740 of determining that the search query correspondsto the question classification. For example, in one or more embodiments,the act 740 includes determining that the search query corresponds tothe question classification for the electronic survey question. In oneor more embodiments, the method 700 includes identifying a searchclassification for the search query and determining that the pluralityof electronic survey responses include information relevant to thesearch query. In one or more embodiments, determining that the pluralityof electronic survey responses includes information relevant to thesearch query by determining that the search classification for thesearch query matches the question classification for the electronicsurvey question associated with the plurality of electronic surveyresponses.

In one or more embodiments, receiving the search query includesreceiving, from the client device, a search query includinguser-composed text. Receiving the search query can also includeanalyzing the user-composed text of the search query to determine a typeof information that the search query is requesting. Receiving the searchquery can also include determining a search classification for thesearch query based on the type of information that the search query isrequesting.

As an alternative or in addition to receiving a search query includinguser-composed text, in one or more embodiments, the method 700detecting, with respect to a graphical user interface of the clientdevice, a user selection of a graphical icon where the graphical iconincludes an identification of a topic and one or more associatedoperators. In addition, the method 700 can include identifying thequestion classification based on the user selection of the graphicalicon.

As further shown in FIG. 7, the method 700 includes an act 750 ofanalyzing the electronic survey responses using the semantics model. Forexample, in one or more embodiments, the act 750 includes, based ondetermining that the search query corresponds to the questionclassification, analyzing the plurality of electronic survey responsesusing the one or more operators of the semantics model to determineresults for the search query. In one or more embodiments, analyzing theplurality of electronic survey responses includes applying the one ormore operators of the semantics model to each of the plurality ofelectronic survey responses to identify information contained within theuser-composed text of the plurality of electronic survey responses.

As further shown in FIG. 7, the method 700 includes an act 760 ofproviding a presentation of results of the analysis of the electronicsurvey responses via a graphical user interface of a client device(e.g., the administrator device 112). For example, in one or moreembodiments, the act 760 includes providing, via a graphical userinterface of a client device, a presentation of results for the searchquery, the results including information identified within the pluralityof electronic survey responses using the one or more operators of thesemantics model. In one or more embodiments, the method 700 includesdetecting a user input with regard to the graphical user interface ofthe client device indicating an incorrect meaning associated with a termor phrase by the semantics model. In addition, in one or moreembodiments, the method 700 includes modifying the semantics model toinclude a different meaning for the term or phrase.

In one or more embodiments, the method 700 includes analyzing theplurality of electronic survey response using the one or more operatorsof the semantics model to determine results for the search query byidentifying a subset of the plurality of electronic survey responsesincluding information identified by the one or more operators of thesemantics model. In addition, in one or more embodiments, the method 700includes providing the presentation of results for the search query byproviding, within the graphical user interface of the client device, alisting of the subset of the plurality of electronic survey responses.

In one or more embodiments, the method 700 includes analyzing theplurality of electronic survey response using the one or more operatorsof the semantics model to determine results for the search query byidentifying instances of words associated with a topic of the electronicsurvey question using the one or more operators of the semantics model.In addition, in one or more embodiments, the method 700 includesproviding the presentation of results for the search query by providing,via the graphical user interface of the client device, a visualizationof the identified instances of words in connection with the topic of theelectronic survey question.

In one or more embodiments, the method 700 additionally includesidentifying multiple question classifications for a given electronicsurvey question. For example, in one or more embodiments, the method 700includes identifying, for the electronic survey question, a secondquestion classification associated with a second semantics model wherethe second semantics model includes one or more additional operators foranalyzing electronic survey responses to the electronic survey question.In addition, in one or more embodiments, the method 700 includesanalyzing the plurality of electronic survey responses using the one ormore operators of the semantics model and the one or more additionaloperators of the second semantics model to identify information withinthe plurality of responses to the electronic survey. In one or moreembodiments, the method 700 further includes identifying the questionclassification and the second question classification based ondetermining that the electronic survey question includes a request formultiple types of information corresponding to the questionclassification and the second question classification.

FIG. 8 illustrates another method 800 for analyzing electronic surveyresponses to one or more electronic survey questions and providing apresentation of analysis results. For example, as shown in FIG. 8, themethod 800 includes an act 810 of receiving electronic survey responsesto a plurality of electronic survey questions. In one or moreembodiments, the act 810 includes receiving, from a plurality ofrespondent devices 108 a-n, a plurality of electronic survey responsesto a plurality of electronic survey questions where the plurality ofelectronic survey responses include user-composed text.

As further shown in FIG. 8, the method 800 includes an act 820 ofreceiving a search query requesting information associated with theelectronic survey responses. As also shown, the method 800 includes anact 830 of determining a question classification and associatedsemantics model associated with an electronic survey question of theplurality of electronic survey questions. In one or more embodiments,the act 830 includes determining, based on the search query, a questionclassification associated with an electronic survey question of theplurality of electronic survey questions where the questionclassification is associated with a semantics model including one ormore operators for analyzing electronic survey responses received inresponse to the electronic survey question. In one or more embodiments,determining the question classification based on the search queryincludes identifying one or more electronic survey questions of theplurality of electronic survey questions including a request for asimilar type of information as a type of information requested by thesearch query.

As shown in FIG. 8, the method 800 includes an act 840 of identifying asubset of the plurality of electronic survey responses corresponding tothe electronic survey question. For example, in one or more embodiments,the act 840 includes identifying a subset of the plurality of electronicsurvey responses corresponding to the electronic survey questionassociated with the question classification.

FIG. 8 also shows that the method 800 includes an act 850 of analyzingthe subset of the electronic survey responses using the semantics model.For example, in one or more embodiments, the act 850 includes analyzingthe subset of the plurality of electronic survey responses using the oneor more operators of the semantics model to determine results for thesearch query.

As shown in FIG. 8, the method 800 includes an act 860 of providing apresentation of results for the search query via a graphical userinterface of a client device. For example, in one or more embodiments,the act 860 includes providing, via a graphical user interface of aclient device, a presentation of results for the search query where theresults include information identified within the subset of theplurality of electronic survey responses using the one or more operatorsof the semantics model.

Embodiments of the present disclosure may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentdisclosure also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. In particular, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices (e.g., any of the media content access devicesdescribed herein). In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein.

Computer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arenon-transitory computer-readable storage media (devices).Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments of the disclosure can comprise at least two distinctlydifferent kinds of computer-readable media: non-transitorycomputer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM,ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM),Flash memory, phase-change memory (“PCM”), other types of memory, otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media tonon-transitory computer-readable storage media (devices) (or viceversa). For example, computer-executable instructions or data structuresreceived over a network or data link can be buffered in RAM within anetwork interface module (e.g., a “NIC”), and then eventuallytransferred to computer system RAM and/or to less volatile computerstorage media (devices) at a computer system. Thus, it should beunderstood that non-transitory computer-readable storage media (devices)can be included in computer system components that also (or evenprimarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. In someembodiments, computer-executable instructions are executed on ageneral-purpose computer to turn the general-purpose computer into aspecial purpose computer implementing elements of the disclosure. Thecomputer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, and the like. The disclosuremay also be practiced in distributed system environments where local andremote computer systems, which are linked (either by hardwired datalinks, wireless data links, or by a combination of hardwired andwireless data links) through a network, both perform tasks. In adistributed system environment, program modules may be located in bothlocal and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloudcomputing environments. In this description, “cloud computing” isdefined as a model for enabling on-demand network access to a sharedpool of configurable computing resources. For example, cloud computingcan be employed in the marketplace to offer ubiquitous and convenienton-demand access to the shared pool of configurable computing resources.The shared pool of configurable computing resources can be rapidlyprovisioned via virtualization and released with low management effortor service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics suchas, for example, on-demand self-service, broad network access, resourcepooling, rapid elasticity, measured service, and so forth. Acloud-computing model can also expose various service models, such as,for example, Software as a Service (“SaaS”), Platform as a Service(“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computingmodel can also be deployed using different deployment models such asprivate cloud, community cloud, public cloud, hybrid cloud, and soforth. In this description and in the claims, a “cloud-computingenvironment” is an environment in which cloud computing is employed.

FIG. 9 illustrates a block diagram of exemplary computing device 900that may be configured to perform one or more of the processes describedabove. One will appreciate that one or more computing devices such asthe computing device 900 may be implemented by the server device(s) 102and/or other devices described above in connection with FIG. 1. As shownby FIG. 9, the computing device 900 can comprise a processor 902, amemory 904, a storage device 906, an I/O interface 908, and acommunication interface 910, which may be communicatively coupled by wayof a communication infrastructure 912. While an exemplary computingdevice 900 is shown in FIG. 9, the components illustrated in FIG. 9 arenot intended to be limiting. Additional or alternative components may beused in other embodiments. Furthermore, in certain embodiments, thecomputing device 900 can include fewer components than those shown inFIG. 9. Components of the computing device 900 shown in FIG. 9 will nowbe described in additional detail.

In one or more embodiments, the processor 902 includes hardware forexecuting instructions, such as those making up a computer program. Asan example and not by way of limitation, to execute instructions, theprocessor 902 may retrieve (or fetch) the instructions from an internalregister, an internal cache, the memory 904, or the storage device 906and decode and execute them. In one or more embodiments, the processor902 may include one or more internal caches for data, instructions, oraddresses. As an example and not by way of limitation, the processor 902may include one or more instruction caches, one or more data caches, andone or more translation lookaside buffers (TLBs). Instructions in theinstruction caches may be copies of instructions in the memory 904 orthe storage 906.

The memory 904 may be used for storing data, metadata, and programs forexecution by the processor(s). The memory 904 may include one or more ofvolatile and non-volatile memories, such as Random Access Memory(“RAM”), Read Only Memory (“ROM”), a solid state disk (“SSD”), Flash,Phase Change Memory (“PCM”), or other types of data storage. The memory904 may be internal or distributed memory.

The storage device 906 includes storage for storing data orinstructions. As an example and not by way of limitation, storage device906 can comprise a non-transitory storage medium described above. Thestorage device 906 may include a hard disk drive (HDD), a floppy diskdrive, flash memory, an optical disc, a magneto-optical disc, magnetictape, or a Universal Serial Bus (USB) drive or a combination of two ormore of these. The storage device 906 may include removable ornon-removable (or fixed) media, where appropriate. The storage device906 may be internal or external to the computing device 900. In one ormore embodiments, the storage device 906 is non-volatile, solid-statememory. In other embodiments, the storage device 906 includes read-onlymemory (ROM). Where appropriate, this ROM may be mask programmed ROM,programmable ROM (PROM), erasable PROM (EPROM), electrically erasablePROM (EEPROM), electrically alterable ROM (EAROM), or flash memory or acombination of two or more of these.

The I/O interface 908 allows a user to provide input to, receive outputfrom, and otherwise transfer data to and receive data from computingdevice 900. The I/O interface 908 may include a mouse, a keypad or akeyboard, a touch screen, a camera, an optical scanner, networkinterface, modem, other known I/O devices or a combination of such I/Ointerfaces. The I/O interface 908 may include one or more devices forpresenting output to a user, including, but not limited to, a graphicsengine, a display (e.g., a display screen), one or more output drivers(e.g., display drivers), one or more audio speakers, and one or moreaudio drivers. In certain embodiments, the I/O interface 908 isconfigured to provide graphical data to a display for presentation to auser. The graphical data may be representative of one or more graphicaluser interfaces and/or any other graphical content as may serve aparticular implementation.

The communication interface 910 can include hardware, software, or both.In any event, the communication interface 910 can provide one or moreinterfaces for communication (such as, for example, packet-basedcommunication) between the computing device 900 and one or more othercomputing devices or networks. As an example and not by way oflimitation, the communication interface 910 may include a networkinterface controller (NIC) or network adapter for communicating with anEthernet or other wire-based network or a wireless NIC (WNIC) orwireless adapter for communicating with a wireless network, such as aWI-FI.

Additionally or alternatively, the communication interface 910 mayfacilitate communications with an ad hoc network, a personal areanetwork (PAN), a local area network (LAN), a wide area network (WAN), ametropolitan area network (MAN), or one or more portions of the Internetor a combination of two or more of these. One or more portions of one ormore of these networks may be wired or wireless. As an example, thecommunication interface 910 may facilitate communications with awireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FInetwork, a WI-MAX network, a cellular telephone network (such as, forexample, a Global System for Mobile Communications (GSM) network), orother suitable wireless network or a combination thereof.

Additionally, the communication interface 910 may facilitatecommunications various communication protocols. Examples ofcommunication protocols that may be used include, but are not limitedto, data transmission media, communications devices, TransmissionControl Protocol (“TCP”), Internet Protocol (“IP”), File TransferProtocol (“FTP”), Telnet, Hypertext Transfer Protocol (“HTTP”),Hypertext Transfer Protocol Secure (“HTTPS”), Session InitiationProtocol (“SIP”), Simple Object Access Protocol (“SOAP”), ExtensibleMark-up Language (“XML”) and variations thereof, Simple Mail TransferProtocol (“SMTP”), Real-Time Transport Protocol (“RTP”), User DatagramProtocol (“UDP”), Global System for Mobile Communications (“GSM”)technologies, Code Division Multiple Access (“CDMA”) technologies, TimeDivision Multiple Access (“TDMA”) technologies, Short Message Service(“SMS”), Multimedia Message Service (“MMS”), radio frequency (“RF”)signaling technologies, Long Term Evolution (“LTE”) technologies,wireless communication technologies, in-band and out-of-band signalingtechnologies, and other suitable communications networks andtechnologies.

The communication infrastructure 912 may include hardware, software, orboth that couples components of the computing device 900 to each other.As an example and not by way of limitation, the communicationinfrastructure 912 may include an Accelerated Graphics Port (AGP) orother graphics bus, an Enhanced Industry Standard Architecture (EISA)bus, a front-side bus (FSB), a HYPERTRANSPORT (HT) interconnect, anIndustry Standard Architecture (ISA) bus, an INFINIBAND interconnect, alow-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture(MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express(PCIe) bus, a serial advanced technology attachment (SATA) bus, a VideoElectronics Standards Association local (VLB) bus, or another suitablebus or a combination thereof.

FIG. 10 illustrates an example network environment 1000 of a surveynetwork 100. Network environment 1000 includes a client device 1006, anda server device 1002 connected to each other by a network 1004. AlthoughFIG. 10 illustrates a particular arrangement of client system 1006,server device 1002, and network 1004, this disclosure contemplates anysuitable arrangement of client device 1006, server device 1002, andnetwork 1004. As an example and not by way of limitation, two or more ofclient device 1006, and server device 1002 may be connected to eachother directly, bypassing network 1004. As another example, two or moreof client device 1006 and server device 1002 may be physically orlogically co-located with each other in whole, or in part. Moreover,although FIG. 10 illustrates a particular number of client devices 1006,server device(s) 1002, and networks 1004, this disclosure contemplatesany suitable number of client devices 1006, server device(s) 1002, andnetworks 1004. As an example and not by way of limitation, networkenvironment 1000 may include multiple client devices 1006, surveydevice(s) 1002, and networks 1004.

This disclosure contemplates any suitable network 1004. As an exampleand not by way of limitation, one or more portions of network 1004 mayinclude an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a cellular telephone network, or a combinationof two or more of these. Network 1004 may include one or more networks1004.

Links may connect client device 1006, and server device 1002 tocommunication network 1004 or to each other. This disclosurecontemplates any suitable links. In particular embodiments, one or morelinks include one or more wireline (such as for example DigitalSubscriber Line (DSL) or Data Over Cable Service Interface Specification(DOCSIS)), wireless (such as for example Wi-Fi or WorldwideInteroperability for Microwave Access (WiMAX)), or optical (such as forexample Synchronous Optical Network (SONET) or Synchronous DigitalHierarchy (SDH)) links. In particular embodiments, one or more linkseach include an ad hoc network, an intranet, an extranet, a VPN, a LAN,a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion ofthe PSTN, a cellular technology-based network, a satellitecommunications technology-based network, another link, or a combinationof two or more such links. Links need not necessarily be the samethroughout network environment 1000. One or more first links may differin one or more respects from one or more second links.

In particular embodiments, client device 1006 may be an electronicdevice including hardware, software, or embedded logic components or acombination of two or more such components and capable of carrying outthe appropriate functionalities implemented or supported by clientdevice 1006. As an example and not by way of limitation, a client device1006 may include any of the computing devices discussed above inrelation to one or more embodiments described herein. A client device1006 may enable a network user at client device 1006 to access network1004. A client device 1006 may enable its user to communicate with otherusers at other client systems.

In particular embodiments, client device 1006 may include a web browser,such as MICROSOFT INTERNET EXPLORER, GOOGLE CHROME, or MOZILLA FIREFOX,and may have one or more add-ons, plug-ins, or other extensions, such asTOOLBAR or YAHOO TOOLBAR. A user at client device 1006 may enter aUniform Resource Locator (URL) or other address directing the webbrowser to a particular server (such as server, or a server associatedwith a third-party system), and the web browser may generate a HyperText Transfer Protocol (HTTP) request and communicate the HTTP requestto server. The server may accept the HTTP request and communicate toclient device 1006 one or more Hyper Text Markup Language (HTML) filesresponsive to the HTTP request. Client device 1006 may render a webpagebased on the HTML files from the server for presentation to the user.This disclosure contemplates any suitable webpage files. As an exampleand not by way of limitation, webpages may render from HTML files,Extensible Hyper Text Markup Language (XHTML) files, or ExtensibleMarkup Language (XML) files, according to particular needs. Such pagesmay also execute scripts such as, for example and without limitation,those written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, combinationsof markup language and scripts such as AJAX (Asynchronous JAVASCRIPT andXML), and the like. Herein, reference to a webpage encompasses one ormore corresponding webpage files (which a browser may use to render thewebpage) and vice versa, where appropriate.

In particular embodiments, server device 1002 may include a variety ofservers, sub-systems, programs, modules, logs, and data stores. Inparticular embodiments, server device 1002 may include one or more ofthe following: a web server, action logger, API-request server,relevance-and-ranking engine, content-object classifier, notificationcontroller, action log, third-party-content-object-exposure log,inference module, authorization/privacy server, search module,advertisement-targeting module, user-interface module, user-profilestore, connection store, third-party content store, or location store.Server device 1002 may also include suitable components such as networkinterfaces, security mechanisms, load balancers, failover servers,management-and-network-operations consoles, other suitable components,or any suitable combination thereof.

In particular embodiments, server device 1002 may include one or moreuser-profile stores for storing user profiles. A user profile mayinclude, for example, biographic information, demographic information,behavioral information, social information, or other types ofdescriptive information, such as work experience, educational history,hobbies or preferences, interests, affinities, or location. Interestinformation may include interests related to one or more categories.Categories may be general or specific.

The foregoing specification is described with reference to specificexemplary embodiments thereof. Various embodiments and aspects of thedisclosure are described with reference to details discussed herein, andthe accompanying drawings illustrate the various embodiments. Thedescription above and drawings are illustrative and are not to beconstrued as limiting. Numerous specific details are described toprovide a thorough understanding of various embodiments.

The additional or alternative embodiments may be embodied in otherspecific forms without departing from its spirit or essentialcharacteristics. The described embodiments are to be considered in allrespects only as illustrative and not restrictive. The scope of theinvention is, therefore, indicated by the appended claims rather than bythe foregoing description. All changes that come within the meaning andrange of equivalency of the claims are to be embraced within theirscope.

We claim:
 1. A method comprising: receiving a search query requestinginformation from a collection of individual user-composed textinstances, wherein each individual user-composed text instance isassociated with at least one semantics model; determining a searchclassification for the search query based on content of the searchquery; identifying a subgroup of individual user-composed text instancesfrom the collection of individual user-composed text instances based ondetermining that the subgroup of individual user-composed text instancesrelate to the search classification for the search query; analyzing thesubgroup of individual user-composed text instances based on the contentof the search query and based on semantics models associated with thesubgroup of individual user-composed text instances; and providing, viaa graphical user interface of a client device, a presentation of theresults for the search query comprising information identified withinthe plurality of electronic survey responses using the semantics modelsassociated with the subgroup of individual user-composed text instances.2. The method of claim 1, further comprising generating the at least onesemantics model to associate with each individual user-composed textinstance by identifying the one or more operators that identify one ormore types of information contained within each individual user-composedtext instance.
 3. The method of claim 2, wherein the one or more typesof information comprise one or more of: opinions, recommendations, orquestions.
 4. The method of claim 1, wherein the collection ofindividual user-composed text instances comprises a plurality ofuser-composed digital survey responses.
 5. The method of claim 1,wherein receiving a search query requesting information from acollection of individual user-composed text instances comprisesreceiving a natural language sentence.
 6. The method of claim 5, whereindetermining a search classification for the search query based oncontent of the search query comprises analyzing the natural languagesentence to determine if the search query is requesting informationrelated to opinions, recommendations or questions.
 7. The method ofclaim 1, further comprising: determining a first group of words relatedto positive opinions from the information identified within theplurality of electronic survey responses using the semantics modelsassociated with the subgroup of individual user-composed text instances;determining a second group of words related to negative opinions fromthe information identified within the plurality of electronic surveyresponses using the semantics models associated with the subgroup ofindividual user-composed text instances; and wherein the presentation ofthe results for the search query further comprises a presentation of thefirst group of words and the second group of words.
 8. A non-transitorycomputer readable storage medium storing instructions thereon that, whenexecuted by at least one processor, cause a computing device to: receivea search query requesting information from a collection of individualuser-composed text instances, wherein each individual user-composed textinstance is associated with at least one semantics model; determine asearch classification for the search query based on content of thesearch query; identify a subgroup of individual user-composed textinstances from the collection of individual user-composed text instancesbased on determining that the subgroup of individual user-composed textinstances relate to the search classification for the search query;analyze the subgroup of individual user-composed text instances based onthe content of the search query and based on semantics models associatedwith the subgroup of individual user-composed text instances; andprovide, via a graphical user interface of a client device, apresentation of the results for the search query comprising informationidentified within the plurality of electronic survey responses using thesemantics models associated with the subgroup of individualuser-composed text instances.
 9. The non-transitory computer readablestorage medium of claim 8, further comprising instructions that, whenexecuted by the at least one processor, cause the computing device togenerate the at least one semantics model to associate with eachindividual user-composed text instance by identifying the one or moreoperators that identify one or more types of information containedwithin each individual user-composed text instance.
 10. Thenon-transitory computer readable storage medium of claim 9, wherein theone or more types of information comprise one or more of: opinions,recommendations, or questions.
 11. The non-transitory computer readablestorage medium of claim 8, wherein the collection of individualuser-composed text instances comprises a plurality of user-composeddigital survey responses.
 12. The non-transitory computer readablestorage medium of claim 8, wherein receiving a search query requestinginformation from a collection of individual user-composed text instancescomprises receiving a natural language sentence.
 13. The non-transitorycomputer readable storage medium of claim 12, wherein determining asearch classification for the search query based on content of thesearch query comprises analyzing the natural language sentence todetermine if the search query is requesting information related toopinions, recommendations or questions.
 14. A system comprising: atleast one processor; and a non-transitory computer readable storagemedium storing instructions thereon that, when executed by the at leastone processor, cause the system to: receive a search query requestinginformation from a collection of individual user-composed textinstances, wherein each individual user-composed text instance isassociated with at least one semantics model; determine a searchclassification for the search query based on content of the searchquery; identify a subgroup of individual user-composed text instancesfrom the collection of individual user-composed text instances based ondetermining that the subgroup of individual user-composed text instancesrelate to the search classification for the search query; analyze thesubgroup of individual user-composed text instances based on the contentof the search query and based on semantics models associated with thesubgroup of individual user-composed text instances; and provide, via agraphical user interface of a client device, a presentation of theresults for the search query comprising information identified withinthe plurality of electronic survey responses using the semantics modelsassociated with the subgroup of individual user-composed text instances.15. The system of claim 14, further comprising instructions that, whenexecuted by the at least one processor, cause the system to generate theat least one semantics model to associate with each individualuser-composed text instance by identifying the one or more operatorsthat identify one or more types of information contained within eachindividual user-composed text instance.
 16. The system of claim 15,wherein the one or more types of information comprise one or more of:opinions, recommendations, or questions.
 17. The system of claim 14,wherein the collection of individual user-composed text instancescomprises a plurality of user-composed digital survey responses.
 18. Thesystem of claim 14, wherein receiving a search query requestinginformation from a collection of individual user-composed text instancescomprises receiving a natural language sentence.
 19. The system of claim14, wherein determining a search classification for the search querybased on content of the search query comprises analyzing the naturallanguage sentence to determine if the search query is requestinginformation related to opinions, recommendations or questions.
 20. Thesystem of claim 14, further comprising instructions that, when executedby the at least one processor, cause the system to: determine a firstgroup of words related to positive opinions from the informationidentified within the plurality of electronic survey responses using thesemantics models associated with the subgroup of individualuser-composed text instances; determine a second group of words relatedto negative opinions from the information identified within theplurality of electronic survey responses using the semantics modelsassociated with the subgroup of individual user-composed text instances;and wherein the presentation of the results for the search querycomprises the first group of words and the second group of words.